Stanford Center for Education 


CEDG | teyAnnes CEPA Working Paper No. 15-10 


Changing Distributions: How Online College Classes Alter 
Student and Professor Performance 


AUTHORS ABSTRACT 
Eric Bettinger Online college courses are a rapidly expanding feature of higher education, yet little research 
Stanford University identifies their effects. Using an instrumental variables approach and data from DeVry 


; University, this study finds that, on average, online course-taking reduces student learning b 
Lindsay Fox : . 2 @ aay 


ae one-third to one-quarter of a standard deviation compared to conventional in-person classes. 
Stanford University 


Taking a course online also reduces student learning in future courses and persistence in 
Susanna Loeb college. Additionally, we find that student performance is more variable online than in 
SENSE ON eTSIEY traditional classrooms but that individual professors explain less of the variation in student 


Eric Taylor performance in online settings than in traditional classrooms. 


Harvard University 


VERSION 


October 2015 


Suggested citation: Bettinger, E., Fox, L., Loeb, S., & Taylor, E. (2015). Changing Distributions: How Online 
College Classes Alter Student and Professor Performance (CEPA Working Paper No.15-10). Retrieved 
from Stanford Center for Education Policy Analysis: http://cepa.stanford.edu/wp15-10 


CHANGING DISTRIBUTIONS: HOW ONLINE COLLEGE CLASSES ALTER 
STUDENT AND PROFESSOR PERFORMANCE* 


Eric Bettinger* 
Lindsay Fox 
Susanna Loeb 
Eric Taylor 


Abstract: Online college courses are a rapidly expanding feature of higher education, yet little 
research identifies their effects. Using an instrumental variables approach and data from DeVry 
University, this study finds that, on average, online course-taking reduces student learning by one- 
third to one-quarter of a standard deviation compared to conventional in-person classes. Taking a 
course online also reduces student learning in future courses and persistence in college. 
Additionally, we find that student performance is more variable online than in traditional 
classrooms but that individual professors explain less of the variation in student performance in 
online settings than in traditional classrooms. 


Total word count: 6,780 


JEL No. 121, 123 


* We greatly appreciate the support of DeVry University, especially Aaron Rogers, Ryan Green, and Earl 
Frischkorn. We also thank seminar participants at Brigham Young University, CESifo, Mathematica Policy 
Research, Stanford University, University of Michigan, University of Stavanger, University of Texas — Austin, 
University of Uppsala, and University of Virginia for helpful discussions and comments. Financial support was 
provided by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B090016 to 
Stanford University. The views expressed and any mistakes are those of the authors. 

* Corresponding author. Address: Center for Education Policy, 520 Galvez Mall, Stanford, CA 94305. Phone: 650- 
736-7727. Email: ebettinger@stanford.edu. 


Online college courses are a rapidly growing feature of higher education. One out of 
three students now takes at least one course online during their college career, and that share has 
increased threefold over the past decade (Allen and Seaman 2013). The promise of cost savings, 
partly though economies of scale, fuels ongoing investments in online education by both public 
and private institutions (Deming et al. 2015). Non-selective and for-profit institutions, in 
particular, have aggressively used online courses. Yet there is little systematic evidence about 
how online classes affect student outcomes. Some studies have investigated the effects of online 
course-taking using distance-to-school instruments or fixed effects (e.g. Hart, Friedmann, and 
Hill 2014, Xu and Jaggars 2013), but in those studies, it is not clear if other aspects of the class 
changed between the in-person and online settings. We provide a clean counterfactual whereby 
we can isolate just the difference in course delivery format. In this paper we estimate the effects 
of online course-taking on student achievement in the course and persistence and achievement in 
college after the course. We examine both the mean difference in student achievement between 
online and in-person courses and how online courses change the variance of student 
achievement. We give specific attention to professors’ contributions to student outcomes, and 
how teaching online changes those contributions. 

Our empirical setting has three salient, advantageous features: an intuitive, straightforward 
counterfactual for each online course; plausibly-exogenous variation in whether students take a 
given course online or in-person; and both at a substantial operating scale. The combination of 
these three features has not been possible in prior work. We study students and professors at DeVry 
University, a for-profit college with an undergraduate enrollment of more than 100,000 students, 
80 percent of whom are seeking a bachelor’s degree. The average DeVry student takes two-thirds 
of his or her courses online. The remaining one-third of courses are conventional in-person classes 
held at one of DeVry’s 102 physical campuses.' The data for this paper cover more than four years 
of DeVry operations, including 230,000 students observed in an average of 10 courses each. 

DeVry University’s approach to online education creates an intuitive, clear counterfactual. 
Each DeVry course is offered both online and in-person, and each student enrolls in either an 


online section or an in-person section. Online and in-person sections are identical in most ways: 


' Our data include 102 campuses. DeVry opens and closes campuses each year, and hence, our number may differ 
from the current number of active campuses. 


both follow the same syllabus and use the same textbook; class sizes are the same; both use the 
same assignments, quizzes, tests, and grading rubrics. The contrast between online and in-person 
sections is primarily the mode of communication. In online sections all interaction—lecturing, 
class discussion, group projects—occurs in online discussion boards, and much of the professor’s 
“lecturing” role is replaced with standardized videos. In online sections participation is often 
asynchronous while in-person sections meet on campus at scheduled times. 

While DeVry students self-sort across online and in-person sections, we use an 
instrumental variables strategy which limits identification to variation arising from where (physical 
campus) and when (academic term) the university offers a course in-person. In a given term, a 
student can choose to take a course in-person, instead of online, if the course is offered at her local 
campus.” Courses are always offered online, but each local campus’ offerings vary from term to 
term. This temporal and spatial variation in in-person offerings is the basis for our instruments. 

Specifically, our first instrument is a simple indicator if the course was offered in-person 
at the student’s nearest campus in any of the current, prior, or following term.’ The identifying 
assumption (exclusion restriction) is that the local campus’ decision to offer an in-person section 
only affects student outcomes by inducing more (fewer) students to take the course in-person. Our 
estimates would be biased, for example, if higher-skilled students demand more in-person 
offerings, and the university’s decisions respond to that demand. Our second and preferred 
instrument, partly in response to this threat, is the interaction between the first instrument and the 
distance between the student’s home and nearest campus. Only this interaction is excluded in the 
second stage; we include the main effects of in-person availability (the first instrument) and 
distance to campus.‘ The pattern of results is similar for either instrument. 

Our estimates provide evidence that online courses do less to promote student learning and 
progression than do in-person courses for students at the margin. Taking a course online reduces 
student achievement by about one-quarter to one-third of a standard deviation, as measured by 
course grades, and reduces the probability of remaining enrolled by three to ten percentage points 
(over a base of 68 percent). Taking a course online also reduces student grade point average in the 


next term by more than one-tenth of a standard deviation. Additionally, we find that student 


2 DeVry divides its academic calendar into six eight-week terms. 

3 As described in Section I, we further limit variation to within-course, -local-campus, and —term; and control for 
prior student academic achievement among other things. 

4 This interaction-instrument approach was first suggested by Card (1995). 
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achievement outcomes are more variable in online classes. The variation of course grades increases 
by as much as one-fifth in online sections, and the variance in student grade point average in the 
next term increases by more than one-tenth. By contrast, we find that the variation in professors’ 
contributions to student achievement and persistence are smaller in online classes than in-person 
classes. 

Several plausible mechanisms could lead students to perform differently in online college 
classes. Online courses substantially change the nature of communication between students, their 
peers, and their professors. First, on a practical dimension, students in online courses can 
participate at any hour of the day from any place. That flexibility could allow for better allocation 
of students’ time and effort. That same flexibility could, however, create a challenge for students 
who have not yet learned to manage their own time. Second, online asynchronous interactions 
change the implicit constraints and expectations on academic conversations. For example, a 
student does not need to respond immediately to a question her professor asks, as she would in a 
traditional classroom, instead she can take the time she needs or wants to consider her response. 
More generally, students may feel less oversight from their professors and less concern about their 
responsibilities to their professors or classmates in the online setting.’ In the standard principal- 
agent problem, effort by the agent falls as it becomes less visible to the principal, and where it 
becomes more difficult for professors to observe student engagement and effort, student outcomes 
may worsen (Jensen and Meckling 1976). Third, the role of the professor is quite different in online 
classes. Notably, eliminating differences in professors’ lectures through standardized video 
lectures should reduce the between-professor variation in student outcomes. Indeed such 
standardization is a necessary condition for the economies of scale promised by online education. 
However, how professors choose to use the time saved by not lecturing could expand that between- 
professor variation. Evidence from K-12 schools consistently shows large and important between- 
teacher variation in student outcomes (Jackson, Rockoff, and Staiger 2014, Rivkin, Hanushek, and 
Kain 2005), yet the academic literature largely ignores such variation in higher education (see 
exceptions by Carrell and West 2010, Bettinger and Long 2010). 

All of these potential mechanisms suggest heterogeneous effects of online classes from 
student to student. Students’ skills, motivations, and work- and home-life contexts may be better 


or worse served by the features of online classes. While some existing work estimates average 


5 In a separate paper, we study peer effects in DeVry’s online college classes Bettinger, Loeb, and Taylor (2015) . 
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effects, few studies have assessed differences across students, and none that we know of 
systematically assesses differential effects due to different course characteristics (e.g. different 
professors). 

Our research contributes to three strands of literature. First, the study provides significant 
evidence of the impact of online courses in the for-profit sector across hundreds of courses. 
Existing evidence with much smaller samples shows negative effects on course completion and 
course grades at community colleges (Xu and Jaggars 2013, Hart, Friedmann, and Hill 2014, Xu 
and Jaggars 2014, Streich 2014b, a) and student exam scores at public four-year institutions 
(Figlio, Rush, and Yin 2013, Bowen et al. 2014).° Students who attend for-profit colleges are 
distinct from other student populations in higher education, and our estimates are the first to focus 
on the impact of online courses in this population. Furthermore, it is possible that the comparison 
between online and in-person courses in the previous studies is confounded by differences in 
syllabi or textbooks, whereas our study presents a clean comparison in which the only difference 
in the courses is the mode of communication. 

Second, our paper contributes to the small literature on professor contributions to student 
outcomes. Work by Carrell and West (2010) examines between-professor variation at the Air 
Force Academy, and finds somewhat less variation than similar estimates in primary and 
secondary schools. We build on this finding demonstrating that at a much less selective institution, 
the between-professor variance is higher than what has previously been found in K-12 and higher 
education settings. We also compare the between-professor variation across online and in-person 
settings and show that the variance is lower in online settings. 

Third, our paper adds to the new and growing literature on private for-profit colleges and 


universities. Research on DeVry University and its peers is increasingly important to 


® Xu and Jaggars (2013) use a distance-to-school instrument for Washington state community colleges and find 
online courses cause a 7 percentage point drop in completion as well as reduction in grades. Their subsequent work 
in 2014 finds that these effects are strongest among young males. Streich (2014b), using an instrument of the percent 
of seats for a course offered online, finds negative effects on the probability of passing of 8.3 percentage points. 
Hart, Friedman and Hill (2014) study California community colleges using individual and course fixed effects and 
also find negative effects of approximately the same size (8.4 percentage points) on course completion. The only 
indication of positive effects comes from Streich (2014a) that finds some evidence of positive employment effects 
though largely in years immediately following initial enrollment when student may still be enrolled. Online courses 
may provide greater opportunities for students to work. 

Working at a public four-year institution, Figlio et al. (2013) randomly assigned students in an economics 
course to take the course either online or in-person and found negative effects of the online version on exam scores 
especially for male, Hispanic and lower-achieving students. Bowen et al. (2014) also study students at four-year 
public institutions, comparing online and hybrid courses. They find no differences in student outcomes. 


=) 


understanding American higher education broadly. The for-profit share of college enrollment and 
degrees is large: nearly 2.4 million undergraduate students (full-time equivalent) enrolled at for- 
profit institutions during the 2011-12 academic year, and the sector granted approximately 18 
percent of all associate degrees (Deming, Goldin, and Katz 2012). Additionally, the sector serves 
many non-traditional students who might be a particularly important focus for policy. Thirty 
percent of for-profit college students have previously attended college and 25 percent having 
attended at least two other institutions before coming to the for-profit sector. Approximately 40 
percent of all for-profit college students transfer to another college (Swail 2009). DeVry University 
is one of the largest and most nationally representative for-profit colleges, making it an ideal setting 
to study this distinct and important group of students. 

While the results suggest that students taking a course online do not perform as well as 
they would have taking the same course in a conventional in-person class, our results have 
limitations. First, a full welfare analysis of online college courses is not possible. Notably, online 
offerings make college courses available to individuals who otherwise would not have access. Our 
estimates are based on students who could take a course in-person or online, and we cannot 
quantify the extent of this access expansion in our setting. Second, we study an approach to online 
courses that is common today, but online approaches are developing rapidly. Further development 
and innovation could alter the results. 

The remainder of the paper is organized as follows. Section I describes the setting, data, 
and approach to estimation. Section II presents our results, and Section III discusses the paper 


including making some conclusions based on our analysis. 


I. Empirical Setting and Methods 


We address three empirical research questions in this paper: 
1. How does taking a course online, instead of in a conventional in-person class, affect 
average student academic outcomes: course completion, course grades, persistence in 


college, and later grades? 


2. How does taking a course online affect the variance of student outcomes? 


3. How does teaching a course online affect the variance of professor performance, as 


measured by professors’ contributions to student outcomes? 


In this section we first describe the specific students, professors, and courses on which we have 


data, and second describe our econometric approach. 


A. Data and Setting 


We study undergraduate students and their professors at DeVry University. While DeVry 
began primarily as a technical school in the 1930s, today 80 percent of the University’s students 
are seeking a bachelor’s degree, and most students major in business management, technology, 
health, or some combination. Two-thirds of undergraduate courses occur online, and the other third 
occur at one of over 100 physical campuses throughout the United States. In 2010 DeVry enrolled 
over 130,000 undergraduates, or about 5 percent of the for-profit college market, placing it among 
the largest for-profit colleges in the United States. 

DeVry provided us with data linking students to their courses for all online and in-person 
sections of all undergraduate courses from Spring 2009 through Fall 2013. These data include 
information on over 230,000 students in more than 168,000 sections of 750 different courses. 
About one-third of the students in our data took courses both online and in-person. Only the data 
from Spring 2012 through Fall 2013 contain information on professors, so part of our analysis is 
limited to this group of approximately 78,000 students and 5,000 professors. In this sub-sample 12 
percent of professors taught both online and in-person classes. Table I describes the sample. Just 
under half of the students are female and average approximately 31 years of age, though there is 
substantial variability in age. Students in online courses are more likely to be female (54 percent 


vs. 35 percent) and older (33.0 years vs. 28.4 years). 


[Table I here] 


The focus of this paper is on the levels and variance of student outcomes. The data provide 
a number of student outcomes including course grades, whether the student withdrew from the 
course, whether the student was enrolled during the following semester and how many units he or 


she attempted, whether the student was enrolled one year later and the number of units attempted 


¢ 


in that semester. Ideally, we would like to know how much students learn in each course the take 
(whether online or in-person); our imperfect measure of learning is course grade. In many higher 
education institutions course grades are subject to professor discretion, and professors may 
exercise that discretion differently in online and in-person classes. That discretion is a 
consideration in this paper, but DeVry’s grading process permits less discretion than the typical 
selective college or university. For each course, DeVry’s professors are asked to follow a common 
rubric for evaluating individual assignments and assigning grades. In many cases quizzes and tests 
are standardized across sections, whether online or in-person. Additionally, alongside course 
grades both for the target course and in future courses, we present results for persistence—a 
consequential outcome for students seeking a degree and one not influenced by professor 
subjectivity. 

As shown in Table I, the average grade was 2.8 (approximately B-), on the traditional zero 
(F) to four (A) scale. There is substantial variation in grades: the standard deviation is 1.3 (more 
than a full letter grade). Approximately 23 percent of students failed any given course, while 41 
percent receive an A- or higher in the courses we observe. Over 88 percent were still enrolled at 
DeVry University in the following semester or had completed their degree. Over 69 percent were 
enrolled one year later or had completed their degree. These outcome means are consistently higher 
in the in-person classes than in the online setting.’ These differences could be the result of selection 
bias, and in the next section we discuss our strategies for overcoming the selection bias in 


estimating causal effects of online courses. 
[Table II here] 


While we focus on one large university in the for-profit sector, our results are likely 
generalizable beyond DeVry University. Table II shows average characteristics of students from 
various sectors of education. The first four columns use data from the National Postsecondary 
Student Aid Study from 2012. The final column uses the DeVry data. DeVry has a similar profile 
to other for-profit colleges, though collectively for-profit colleges differ from traditional sectors, 
including two-year colleges, given their focus on non-traditional students and African-American 


students. 


7 This difference remains true when we limit the sample to students who take both online and in-person classes. 
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B. Empirical Methodology 


Effects on the Mean of Student Outcomes—We first estimate the effect of taking a course online, 
instead of a traditional in-person classroom, on the average student’s success in the course and 
persistence in college after the course. We measure student success using course grades. We 
measure student persistence in the subsequent semester and the semester one year later. 
Calculating the difference in means is straightforward, but the decision to take a course in an 
online section or traditional section is likely endogenous—driven by unobservable information 
that could also influence each student’s chances of success in the online versus traditional 
options. Our identification strategy is to use only variation in online versus traditional course 
taking that arises because of idiosyncratic changes over time in each local campus’s in-person 
course offerings. 

Our first student outcome is the grade, y;-z, received by student i in course c during term 
t. Each professor assigns traditional A-F letter grades, which we convert to the standard 0-4 point 
equivalents.* To estimate 6—the mean difference in course grades, y;,;, between students in online 


and traditional classes—we first specify the following statistical model: 
(1) Vict = OOnlineicg + AVi cet + XitB +e + ht + Wocit) + Eict 


In addition to the variable of interest, the Online;,, indicator, specification 1 includes several 
relevant controls: students’ prior grade point average (GPA), ¥;7<¢, measured in all terms before 
term t, observable student characteristics, X;, (gender and age), course fixed effects, 7,, for each 
of the 800+ courses, and a non-parametric time trend, @;, over the 27 terms (4.5 years, May 2009 
through the end of 2013) in our data. Specification 1 also includes fixed effects for each student’s 
“home campus” represented by Wp iz). DeVry University operates over 100 local campuses 
throughout the United States. We assign each student to a home campus, b, based on the physical 


distance between the student’s home address and the local campus addresses; selecting as “home” 


8 An A is 4 points, A- is 3.7, B+ is 3.3, B is 3, etc. 


the campus with the minimum distance.’ By limiting estimation to the within-campus over-time 
variation, we account for fixed differences between local campuses in the scope of course 
offerings. 

Fitting specification 1 by least squares will provide an unbiased estimate of 6 under the 


assumption: 
(2) E[ ict | Online cict) | = Eléice] 


In words, students taking online classes are no more likely to score higher (lower) than students 
taking traditional in-person classes, conditional on included controls. While the available 
controls, especially prior GPA and home campus, are helpful, we remain concerned about 
unobserved student-level determinants of the online versus in-person decision that would bias 
our estimates. 

In response to these concerns we propose an instrumental variables strategy. Specifically, 
we instrument for Online, j¢¢) in specification | with an indicator variable = 1 if student i’s home 
campus b offered course c on campus in a traditional class setting during term t. Combined with 
the home campus fixed effects, Wpcjt), this limits the identifying variation to between-term 
variation within-campuses in the availability of an in-person option for a given course. This 
strategy produces a local average treatment effect (LATE) estimate, where 6 is interpreted as the 
effect of taking a course online for compliers: students who would take a course in-person if it was 
offered at the nearest DeVry campus but otherwise take the course online. 

With this setting “never takers” deserve special consideration. A never taker is a student 
who would always take the given course in-person. Notably, if the course is not offered at their 
home campus in the term they would prefer to take the course, these never takers may wait until 
next term to take the course or take the course earlier than they would prefer in order to find an in- 
person section. This potential inter-temporal behavior motivates a modification to our instrument. 


Specifically, we instrument for Online,(j-¢) in specification 1 with an indicator variable = 1 if 


° In addition to students with missing address data, we exclude all students with international addresses and students 
whose addresses are not within the continental United States. The resulting sample contains 78 percent of the 
universe of undergraduate DeVry students over the time frame. 
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student i’s home campus b offered course c on campus in a traditional class setting during term 
any of terms t,t — 1, ort +1.'° 

In this instrumental variables framework, the identifying assumption (exclusion restriction) 
is: The local campus’ decision to offer an in-person section only affects student outcomes by 
inducing more (fewer) students to take the course in-person. Our estimates would be biased, for 
example, if higher-skilled students demand more in-person offerings, and the university’s 
decisions respond to that demand. This assumption seems plausible but is untestable. As a 
robustness check, we report results using two alternative instruments. 

Our second and preferred instrument is the interaction between the first instrument and the 
distance between the student’s home and nearest campus. The logic here is that a student’s local 
campus offerings should be more influential for students who live learner their local campus. Only 
the interaction is excluded in the second stage; we include the main effects of in-person availability 
(the first instrument) and distance to campus in the second stage. This kind of interaction 
instrument was first proposed by Card (1995). While further limiting the identifying variation, the 
advantage of this instrument is that it allows us to control directly for in-person availability and 
distance from campus. 

For completeness, and comparison to prior papers, we also report results using a third 
instrument: the simple distance between the student’s home and nearest campus, without regard to 
course offerings at that campus. The advantages and disadvantages of such distance instruments 
have been long discussed (e.g. Card 2001). We focus in this paper on results from the first and 
second instruments. 

In addition to student grade points, we use the same specification and strategy to estimate 
6 for other student outcomes. First, we examine whether the effect on grade points is concentrated 
in a particular part of the grade distribution. We replace grade points with binary outcome 
variables: receiving an A- or higher, a B- or higher, a C- or higher, and simply passing the course. 
These results are provided in the appendix. Second, we turn to student persistence outcomes. We 
estimate the difference in the probability of enrolling at DeVry in the very next semester, and the 


difference in the number of units a student attempts the next semester. We repeat the same analysis 


'0 This instrument assumes that “never taker’ students do not shift their course taking by more than one term before 
or after their preferred term. In results not presented we find that expanding the instrument to include t + 2 ort +3 
does not change the pattern of results. 
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for enrollment during the semester one year after term t.11 Results for the extensive margin of 
persistence are presented here; the intensive margin results are provided in the appendix. Lastly, 


we repeat the analysis using a student’s GPA in the next term as the outcome. 


Effects on the Variance of Student Outcomes—Our second objective is to estimate the effect of 
taking a course online, instead of a traditional in-person classroom, on the variance of student 
outcomes. If students were randomly assigned to take a course online or in-person, the parameter 
of interest could be estimated straightforwardly by 

P = var (Vict | Onlinejce = 1) — var(Yict | Onlineice = 0). 

We estimate the variance effect, y, using an instrumental variables approach that builds 
on our approach for estimating the mean effect. First, we obtain the fitted residuals, €;.;, after 
two-stage least squares estimation of specification 1. Second, we repeat the identical two-stage 
least squares regression used to estimation specification 1, except that the dependent variable, 
Vict» is replaced with the squared residuals 7... This is analogous to the familiar steps in FGLS 
or tests for heteroskdasticity. Unbiased estimates of y require the same assumptions as the IV 
estimate for mean effects. 

As an alternative approach we estimate y using a linear mixed model extension of 


specification 1: 


(3) Vict = OOnlineice + AVi ret + Xith + Wo + ht + Wocit) 


+a? Onlinejce + af (1 — Onlinejce) + Eicts 


6 2 
; . a; ~ O oO 
with the assumption a ~N de j | a aa b. 
at 


i 0 QO qT 
The new terms a? and a? are student random effects, specific to online and traditional class 
settings respectively. We estimate 3 by maximum likelihood and obtain 7 = (6%o ss Gr). We 


limit the estimation sample to students who have taken at least one course online and one course 


‘| DeVry’s academic calendar divides the year into six terms, with two consecutive terms equivalent to a semester in 
a more traditional calendar. Following the University’s approach, we define “enrollment the next semester” as 
enrollment during either term t + 1 or t + 2 or both. We define “enrollment one year later” as enrollment during 
either term t + 6 or t + 7 or both. 


12 


in-person. This approach does not directly address student selection, but it has the benefit of 
estimating the variance of student outcomes in online and traditional classes directly rather than 
just estimating the difference in variance (as in the first approach). Furthermore, to partly address 
selection we replace the dependent variable, y;,;, in specification 3 with the residual from 2SLS 


estimation of specification 1, &;,;. 


Effects on the Variance of Professor Contributions to Student Outcomes—Finally, we estimate 
the effect of teaching a course online on the variance of professors’ contributions to student 
outcomes. These estimates provide important context for understanding potential mechanisms 
behind the other effects we estimate. The estimates are also of interest because they provide 
information on how professor job performance varies, and how that variability is affected by 
moving a class online. 

We estimate the difference in the variance of professor effects using a linear mixed model 
approach analogous to the specification in 3. In this case we replace the student random effects 


with random effects for professors, (ic), and for course sections, O¢(jct). 


(4) Vict = OOnlinej, + f Viret) +XpB +e + hp + Wocit) 
+ Uv ict) Onlineice + Hiticty A — Online;-+) 


+O ict Onlineice + Oxict)(1 — Onlineict) + €ict, 


2 
a Th 0,0 ,,T Ge 
with the assumption H ; ~N( a | ee Lu ; ) 
as Ze 0 0 Ogo 
T 
65 6 0 0 Co gr Oar 


The section random effects, @s(jc¢¢), capture shocks common to all students in student i’s class, s, 
but not common to all classes taught by professor j. We estimate 4 by maximum likelihood and 
4 _ (a2 _ -2 
obtain A = (62, 61). 
Though we do not have an instrument for whether a professor teaches online or in-person, 


we can use the instrumental variables approach to help address student selection. Specifically, we 


replace the dependent variable, y;,;, in specification 4 with the residual from 2SLS estimation of 
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specification 1, €;,,. In both approaches, we show results (1) for all professors and (ii) limiting the 
estimation sample to professors who have taught at least one course online and one course in- 
person. Additionally, the estimation sample is limited to nine terms from May 2012 forward; the 


data system used before May 2012 did not record the professor for each course section. 


II. Results 


A. Effects on Average Student Outcomes 


For the average student, taking a course online, instead of in a traditional in-person 
classroom setting, reduces student learning, as measured by course grades, and lowers the 
probability of persistence in college. Table III reports our instrumental variables estimates. (The 
ordinary least squares estimates shown in Panel A of Appendix Table A.1 are similar.) These 
estimates are consistent with the negative effects reported in other settings and papers. 
Nevertheless, our estimates provide new information on an important but unstudied population — 
students at private for-profit colleges and universities — and they are based on more convincingly 


exogenous variation. 


[Table II here] 


Panel A of Table III identifies the effect of taking a course online instead of in-person by 
instrumenting for taking a course online with an indicator = 1 if the student’s local campus offered 
the course in-person either during the term the student took the course or the term just before or 
after. The coefficient on the excluded instrument in the first stage is highly statistically significant, 
indicating that when the course is offered in-person, students are 17.5 percentage points less likely 
to take the class online. The local average treatment effect of taking a course online versus in- 
person on achievement is a 0.38 grade point drop in course grade, approximately a 0.28 standard 
deviation decline. Put differently, complier students taking the course in-person earned roughly a 
B- grade (2.8) on average while their peers in online classes earned a C (2.4). Results presented in 
the appendix suggest this effect is true across the grade distribution, not simply at the top or bottom. 


The estimates for persistence show a 3 percentage point drop in the probability of remaining 
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enrolled (or graduating within) one year later. The complier students taking the course in-person 
had a 69 percent probability of remaining enrolled while their peers in online courses had a 66 
percent probability. Finally, the effect of taking a course online versus in-person on students’ GPA 
next term is a 0.13 grade point drop. 

Panel B shows local average treatment effects using the distance from a student’s home 
address to the nearest DeVry campus as the instrument. The results are larger in magnitude and all 
highly statistically significant, though the source of variation is less convincingly causal. Panel C 
uses our alternative instrument which is an interaction between the indicator for whether the 
student’s local campus offered the course in-person and the distance between the student’s home 
and the local campus. The first stage on this instrument is positive and highly significant, indicating 
that if the course is offered in-person, the probability of taking the course online increases as 
distance from the campus increases. With this strategy, the effect of taking a course online instead 
of in-person is statistically significantly negative. The LATE estimate is a drop of 0.44 grade 
points, or a 0.33 standard deviation decline. The results for persistence indicate an 11 percent drop 
in the probability of being enrolled one year later, and the results for GPA in the next term show a 
drop of 0.17 grade points. 

While our setting is quite different, it is useful to consider other effects in the higher 
education literature. For example, the literature on financial aid often finds that $1000 in financial 
aid increases persistence rates by about three percentage points (Bettinger 2004) and college 


mentorship increases persistence rates by five percentage points (Bettinger and Baker 2013). 


[Table IV here] 


The estimates in Table III are estimates of effects among compliers; in this case compliers 
are students who, for a given course, take the course in-person if it is offered in-person at the 
nearest DeVry campus but otherwise take the course online. The remaining students—the non- 
compliers—include “always online” students who would take the course in question online no 
matter the local in-person offerings, and similarly “always in-person” students. (We assume there 
are no defiers). To get a sense of how the complier students differ from their always online and 


always in-person peers, we can estimate the average observable characteristics of these three 


ite) 


groups.!? Table IV provides these results. Compared to other DeVry undergraduates, compliers 
are older, less likely to be female, and have higher prior GPAs on average. 

In addition to assessing the outcomes in Table III, we also estimate the effects of online 
course taking on a range of other outcomes which are reported in Appendix Table A.1. We find 
that online students are less likely to pass the course (a reduction of 8.4 percentage points), and 
less likely to receive an A minus or above by (a reduction of 12.3 percentage points). After taking 
an online course, the students are 8.9 percentage points less likely to enroll in the following 


semester and, if they do, take fewer credits.'° 
[Table V here] 


We also explore whether the negative impact of taking a course online is heterogeneous by 
student ability, measured by prior grade point average. Table V shows results for course grades 
and persistence from a model in which taking a course online is interacted with prior GPA. The 
results indicate that the effect of taking a course online is less negative for students with higher 
prior GPAs. For example, the point estimates from Panel C imply that a one point increase in prior 
GPA increases the impact of taking an online course instead of an in-person course by 0.12 grade 
points. The effect on GPA next term is slightly smaller with a magnitude of 0.11. This pattern 
holds across the full set of outcomes (see Appendix Table A.2). 


B. Effects on the Variance of Student Grades 


In addition to reducing average achievement, online courses also increase the variance of 
student academic performance as measured by course grades both in the target course and during 
the next term. Table VI reports estimates using our three instruments (Rows A-C) as well as the 
maximum-likelihood student-random-effects approach (Rows D-E). When students take a course 


online, instead of in a conventional in-person class, the between-student standard deviation of 


X-yXny—-AXa 


'2 The average value of X among compliers is given by: Xe = where C,N,A denote compliers, never- 


Tc 
takers, and always-takers, respectively, and 7c is the proportion of the sample in a particular group. For these 
calculations, we use the first (binary) instrument based on availability. 
'3 The results are unchanged when we limit the sample to students who are taking the course for the first time. When 


we code withdrawals as a failing grade, the results are consistently larger in magnitude. 
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grades is one-eighth to one-fifth larger. This increase in variance is large but perhaps not surprising 
given our discussion in the introduction of differences between online and in-person practices and 


technologies, and the results on heterogeneity by prior GPA. 


[Table VI here] 


C. Professor Effects on Student Outcomes 


Difference in professor job performance between online and in-person settings is a 
potential mechanism that could generate differences in student performance. Yet while the 
variance of student achievement increases in online classes, we find the opposite for professor 
performance, as measured by professor contributions to student achievement. Table VII provides 
these results: Panel A reports the estimates from the simple estimation of equation 4, and panels 
B-D show estimates of equation 4 where the dependent variable is the residual from the student 
IV model. 

When estimated using the full sample of professors (Columns 1-2) we find large, 
statistically-significant differences in the variance of professor contributions to student 
achievement. In online classes, between-professor variation in student grades is significantly 
smaller (a standard deviation of 0.24 grade points) than between-professor variation in in-person 
classes (0.32 grade points). Much smaller but still statistically significant differences exist when 
the outcome is GPA next term (0.03 and 0.09 points, respectively). However, these results remain 
subject to selection bias from professors sorting across online and in-person settings. As a first- 
order response to professor selection concerns, we limit the estimation sample to only those 
professors we observe teaching both online and in-person. In this limited sample (Columns 3-4), 
the differences between the variance in professor contributions to student grades in online and in- 
person settings persists and is only slightly smaller in magnitude (0.27 grades points in in-person 
versus 0.23 grade points in online). There is no statistical difference in the variance of professor 
contributions between online and in-person settings in terms of student GPA in the next term. 

Across the different models, the estimate of the standard deviation of professor 
contributions ranges from 0.24 to 0.32 when the outcome is course grade. In student standard 


deviation terms, these are roughly 0.18 and 0.24. That places these estimates at the upper end of 
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similar estimates from K-12 teachers, measured with student test scores. And both the online and 
in-person professor differences are noticeably larger than estimates from the more-selective, in- 


person Air Force Academy (Carrell and West 2010). 


[Table VII here] 


III. Discussion and Conclusion 


This study is the first, of which we are aware, to not only estimate the effects of at-scale 
online courses but to also understand the distributional consequences of online courses at non- 
selective 4-year institutions of higher education or at for-profit institutions, both of which serve a 
substantial proportion of students. In addition, this study uses an instrumental variables strategy 
for addressing selection and estimating causal effects that is arguably more convincing than prior 
approaches to estimating effects of online course taking at scale. Furthermore, in contrast to 
previous studies of online courses, our setting provides a clean counterfactual in which the only 
difference between online and in-person courses is the medium of instructional delivery. All 
other aspects of the course — professor assignment, class size, syllabus, textbooks, etc. — are 
identical across online and in-person courses. 

Our analyses provide evidence that students in online courses perform substantially 
worse than students in traditional in-person courses, and these findings are robust across a 
number of specifications. We also find that the variance of student outcomes increases, driven at 
least in part, by differentially larger negative effects of online course taking for students with 
lower prior GPA. The results are in line with prior studies of online education in showing that in- 
person courses yield better mean outcomes than online courses (e.g. Figlio, Rush, and Yin 2013, 
Streich 2014b). Our estimates of lower grades and lower persistence in online courses are similar 
to those in Xu and Jaggars (2013, 2014) and Hart, Friedman, and Hill (2014). 

While we find that online courses lead to poorer student outcomes, we cannot provide a 
full welfare analysis. Most notably, the existence of an online course option might have enabled 
some students to take college courses that otherwise would not have done so. First, our estimates 
are most appropriately generalized to students who could take a course online or in-person. 


Second, we cannot estimate the extent of this expansion in college course access in our setting. 
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Our estimates are, nevertheless, a critical input to a more complete welfare analysis. 

There are several potential explanations for the increase in student variance. Figlio et al. 
(2013) argues that procrastination in online settings was much more prevalent and led to the 
worsening of outcomes. While we have no measure of procrastination, we find the worsening of 
outcomes is especially large at the bottom of the distribution — students who either withdraw or 
fail the course. These students generally fail to complete the course requirements and may have 
procrastinated in their course responsibilities. We find fewer differences in outcomes at the top 
of the grade distribution suggesting that online courses may not be worse for high achieving 
students. If indeed procrastination is driving many of the results, courses could be redesigned to 
improve student participation. 

As for the change in the variance attributable to professors, the decrease in variance in 
online classes would be both good and bad. It might eliminate extreme outcomes from bad 
instructions, yet it might also stifle good teaching. A number of potential reasons for these 
differences exists. In the online courses, professors use the same syllabus, lectures, and materials, 
and in some cases, even the same scripts. The greater adherence to a script among professors 
surely contributes to a more similar and less varied experience across professors. It may also be 
the case that online interactions are more limited than in-person so there are fewer opportunities 
for professors to influence students. 

Overall, the results — lower student performance, greater student variation, and lower 
professor variation — while not necessarily surprising, provide evidence that online courses are 
not yet as effective as in-person courses. This current state, however, is clearly not a necessary 
end state. The greater homogeneity of online courses points to the potential to scale good courses 
and reduce the variability in experiences that students have within the same course taught by 
different professors in traditional classrooms. Lastly, our results provide a rationale and jumping 
off point for exploring the potential mechanisms that could be driving the results such as student 


effort, procrastination, and professor behavior. 
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Table I—Student Characteristics and Outcomes 


Observations 
Student-course-session 
Students 
Professors 
Courses 
Sections 


Student characteristics 
Female 
Age 


Online 


Student prior academics 
Cumulative GPA 


Failed any course 
Withdrew from any course 


Student course outcomes 
Grade (0-4) 


A- or higher 
B- or higher 
C- or higher 
Passed 


Student persistence outcomes 
Enrolled next semester 
Credits attempted 
next semester 
Enrolled semester one year later 
Credits attempted 
semester one year later 


All Online In-person 
(1) (2) (3) 
2,323,023 1,373,521 949,502 
230,484 184,799 118,041 
5,290 2,590 3,370 
750 559 653 
168,223 63,443 104,780 
0.467 0.545 0.354 
31107 32.986 28.390 
0.591 1 0 
3.027 3.057 2.983 
(0.866) (0.873) (0.853) 
0.234 0.229 0.241 
0.214 0.214 0.212 
2.821 2.798 2.856 
(1.329) CL.357) (1.285) 
0.410 0.413 0.406 
0.696 0.688 0.707 
0.831 0.821 0.845 
0.884 0.875 0.898 
0.882 0.874 0.893 
9.764 9.126 10.652 
(4.657) (4.555) (4.651) 
0.686 0.681 0.693 
7.737 6.899 8.906 
(5.642) (5.392) (5.774) 


Note. Authors' calculations. Means (standard deviations) for DeVry University undergraduate course enrollments 


from May 2009 to November 2013. 
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Table II—Student Characteristics, Institution Types, and DeVry University 


Private 
not-for- 
Public profit 4- Public Private DeVry 
4-year year 2-year __ for-profit University 
(1) (2) (3) (4) (5) 
Age 
23 or younger 69.6 712 49.1 31.6 28.1 
24-39 24.8 19.0 36.4 50.7 52.5 
40+ 5.6 9.9 14.4 17.7 19.4 
Female 5339 56.6 Soe 64.1 44.5 
Race/Ethnicity 
African-American or Black 12.8 13.4 16.4 25.6 25.5 
Asian 6.9 6.9 5.0 29: 4.9 
Hispanic or Latino 13.8 10.1 18.6 18.5 18.7 
Other, More than one 4.4 4.5 4.3 4.6 4.8 
White 62.2 65.1 55.8 48.5 46.1 


Note. Columns 1-4 come from the 2011-12 National Postsecondary Student Aid Study (NPSAS:12) as reported by 
NCES QuickStats. Column 5 is from DeVry administrative data. Race/ethnicity is imputed for 16.9 percent of 


students assuming missing at random. 
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Table IUN—Effect of Taking a Course Online, Instead of In-person, on Student Achievement and 
Persistence (Local Average Treatment Effects) 


Dependent variable 


Course grade 


(A=4...F= Enrolled Next Enrolled One GPA next 
0) Semester Year Later term 
(1) (2) (3) (4) 
A. Instrument = Course available in person at home campus 
Online -0.380*** -0.016** -0.025** -0.134*** 
(0.017) (0.005) (0.009) (0.014) 
First stage coefficient on 
instrument -0.175 -0.167 -0.167 -0.181 
First stage F-statistic 161.99 115.41 115.41 181.46 


B. Instrument = Distance to home campus 


Online -0.572*** -0.08 1*** -0.183*** -0.554*** 
(0.071) (0.020) (0.052) (0.120) 

First stage coefficient on 

instrument 0.001 0.001 0.001 0.001 

First stage F-statistic 19.40 17.13 17.13 18.76 


C. Instrument = Course availability interacted with distance to home campus 


Online -0.442*** -0.089*** -0.106*** -0.173*** 
(0.023) (0.010) (0.014) (0.027) 

First stage coefficient on 

instrument 0.001 0.001 0.001 0.001 

First stage F-statistic 193.02 139.20 139.20 205.32 

Sample mean (st. dev.) for 

dep. var. 2.821 0.882 0.686 2.817 

(1.329) (1.234) 
Number of observations 2,323,023 2,360,645 2,360,645 1,974,831 


Note. The first row of each panel reports the estimated local average treatment effect from a separate two-stage least 
squares regression. Row two reports the coefficient on the excluded instrument from the first stage. The rows at the 
bottom of the table report the sample mean (standard deviation) of the dependent variable. Dependent variables 
described in column headers. The specification includes one endogenous treatment variable, an indicator = | if the 
student took the course online. In panel A, the excluded instrument is a binary variable = 1 if the course was offered 
in-person at the student’s home campus (defined as the nearest campus) in any of the current term, the previous 
term, or the following term. In panel B, the excluded instrument is the distance in miles from the student's home 
address to her home campus. In panel C, the excluded instrument is the interaction between availability in the 
current, previous or following term and distance to home campus. All specifications also included controls for (1) 
prior cumulative grade point average, (ii) an indicator for student gender, (iii) student age, and (iv) separate fixed 
effects for course, term, and home campus. The estimation sample is limited to students who have address 
information. Standard errors allow for clustering within sections. *p<.05, **p<.01, ***p<.001 
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Table [V—Observable Characteristics of Compliers 


Compliers Always Always 


Online In- 

Characteristic Person 

() (2) (3) 
Female 0.45 0.54 0.34 
Age 31.67 32.39 28.01 
Prior GPA 3.10 2.97 3.05 
Prior GPA in online courses 3.06 2.92 218 
Prior GPA in in-person courses 2.99 3.14 3.10 
Proportion of courses taken online 0.43 0.82 0.14 
Proportion repeating a course 0.03 0.06 0.03 
Number of courses taken in first session 1.75 1.77 1.95 
Proportion of courses taken online in first session 0.36 0.82 0.12 
Number of units taken in first session 5.90 5.70 6.32 
Proportion of units taken online in first session 0.36 0.82 0.12 
Proportion taking all online courses in first session 0.31 0.77 0.06 
Proportion taking all in-person courses in first 
session 0.58 0.14 0.81 
Proportion taking a mix of courses in first session 0.11 0.10 0.13 
Proportion taking a course within their major 0.55 0.28 0.56 
Initial Business major 0.46 0.53 0.25 
Initial Computers major 0.36 0.32 0.54 
Initial Health major 0.16 0.14 0.21 
Initial Criminal Justice Major 0.02 0.01 0.01 


Note. Assuming no "defiers", compliers make up 27.4 percent of the sample, always-takers 47.8 percent, and never- 

takers 24.8 percent. The treatment indicator is = | if the student took the course online. The excluded instrument is a 
binary variable = | if the course was offered in-person at the student’s home campus (defined as the nearest campus) 
in any of the current term, the previous term, or the following term. The estimation sample is limited to students who 
have address information: 2,323,023 student-course observations. 
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Table V—Heterogeneity of Effect of Taking a Course Online, Instead of In-person, by Prior 
Achievement 


Dependent variable 


Course grade Enrolled 
(A=4...F= Next Enrolled One GPA next 
0) Semester Year Later term 
(1) (2) 3) (4) 
A. Instrument = Course available in person at home campus 
Online “01 3** -0.252*** -0.365*** -0.506*** 
(0.069) (0.025) (0.038) (0.047) 
Online * prior grade point 
avg. O19 2er° 0.086*** 0.124*** O1332e 
(0.025) (0.009) (0.015) (0.016) 
First stage F-statistic: online 191.299 131.694 131.694 198.406 
First stage F-statistic: 
interaction 105.053 80.627 80.627 121.073 
B. Instrument = distance to home campus 
Online -0.769*** -0.150*** -0.258*** -0.675*** 
(0.051) (0.015) (0.039) (0.087) 
Online * prior grade point 
avg. O134*** 0.050*** 0.0547"* 0.076** 
(0.016) (0.003) (0.009) (0.023) 
First stage F-statistic: online 571.447 372.624 372.624 454.259 
First stage F-statistic: 
interaction 163.017 130.111 130.111 163.866 
C. Instrument = interaction between availability and distance 
Online -0.621*** -0.173*** -0.21 1 *** -0.339%** 
(0.023) (0.008) (0.010) (0.023) 
Online * prior grade point 
avg. O123*** 0.062*** 0.079*** O07 e** 
(0.009) (0.004) (0.005) (0.007) 
First stage F-statistic: online 873.301 669.444 669.444 555.450 
First stage F-statistic: 
interaction 98.329 88.627 88.627 114.106 
Number of observations 2,323,023 2,360,645 2,360,645 1,974,831 


Note. Each column, within panels, reports estimates from a separate two-stage least squares regression. Dependent 
variables described in column headers. The specification includes two endogenous treatment variables, an indicator 
= | if the student took the course online, and the interaction of that indicator and students' cumulative prior grade 
point average. The excluded instruments are a binary variable = | if the course was offered in-person at the student’s 
home campus (defined as the nearest campus) in any of the current term, the previous term, or the following term 
(panel A), the distance in miles from the student’s home address to her home campus (panel B), the interaction of 
availability and distance (panel C); and the interaction of those variables, respectively, with prior GPA. All 
specifications also included controls for (i) prior cumulative grade point average main effect, (ii) an indicator for 
student gender, and (iil) student age, and (iv) separate fixed effects for course, term, and home campus. The 
estimation sample is limited to students who have address information. Standard errors allow for clustering within 
sections. *p<.05, **p<.01, ***p<.001 
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Table VI—Effect of Taking a Course Online, Instead of In-person, on the Variance of Student 


Achievement 
Sample 
st. dev. Online - P-value Student- 
course in-person test course 
grade diff. diff.=0 observations 
(1) (2) (3) (4) 
A. Instrument = Course available in person at home campus 
Dep Var = Course grade 2.821 0.361 0.000 2,323,023 
Dep Var = GPA next term 2.871 0.158 0.260 1,974,831 


B. Instrument = Distance to home campus 
Dep Var = Course grade 2.821 0.534 0.006 2,323,023 
Dep Var = GPA next term 2.871 0.608 0.020 1,974,831 


C. Instrument = Course availability interacted with distance to home campus 


Dep Var = Course grade 2.821 0.566 0.000 2,323,023 

Dep Var = GPA next term 2.871 0.375 0.000 1,974,831 
D. Random effects estimate (all students) 

Dep Var = Course grade 2.821 0.183 0.000 2,323,023 

Dep Var = GPA next term 2.871 0.049 0.000 1,974,831 


E. Random effects estimate (only students who took courses both online and in-person) 
Dep Var = Course grade 2.891 0.344 0.000 1,118,736 
Dep Var = GPA next term 2.844 0.078 0.000 1,050,322 


Note. Column 2 reports the estimated effect of taking a course online on the standard deviation of student course 
grade. The dependent variable scale is 0-4, A= 4 ... F=0. Column | reports the sample standard deviation of course 
grade. 

The estimates in rows A-C are instrumental variables estimates obtained in two steps. Step 1 obtain the fitted 
residuals from the two-stage least squares regression used to estimate the mean effect on course grades. Step 2 
repeat the identical two-stage least squares regression except that the dependent variable course grade is replaced 
with the squared residuals from step 1. Just as in Table III, the 2SLS specification includes one endogenous 
treatment variable, an indicator = | if the student took the course online. The excluded instrument in row A is a 
binary variable = | if the course was offered in-person at the student’s home campus (defined as the nearest campus) 
in any of the current term, the previous term, or the following term. The excluded instrument in row B is distance in 
miles from the student’s home address to her home campus. The excluded instrument in Row C is the interaction of 
availability and distance. All specifications also includes controls for (i) prior cumulative grade point average, (ii) an 
indicator for student gender, (iii) student age, and (iv) separate fixed effects for course, term, and home campus. The 
estimation sample is limited to students who have address information. The test in column 3 allows for error 
clustering within sections. 

The estimates in rows D and E are random effects estimates from a linear mixed model. The specification includes 
a random student effect which is allowed to be different for online and in-person classes. The fixed portion of the 
model includes the same regressors (1)-(iv) as the 2SLS model. The test in column 3 is a likelihood-ratio test where 
the constrained model requires the online and in-person variance parameters to be equal. 
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Table VII—Variance of Professor Effects in Online and In-person Classes 
(Random Effects Estimates) 


A. B. 
All professors Professors who teach 
both 
in-person and online 
Course GPA next Course GPA next 
grade session grade session 
(1) (2) (3) (4) 


A. Dep. Var.: observed values 
Standard deviation of professor effects 


In-person classes 0.316 0.0858 0.270 0.0622 
Online classes 0.243 0.0347 0.232 0.0534 
In-person = online test p-value [0.000] [0.000] [0.002] [0.470] 


B. Dep. Var.: residuals after 2SLS with instrument = Course available in person at home campus 
Standard deviation of professor effects 


In-person classes 0.313 0.0873 0.274 0.0640 
Online classes 0.235 0.0320 0.227 0.0505 
In-person = online test p-value [0.000] [0.000] [0.000] [0.265] 


C. Dep. Var.: residuals after 2SLS with instrument = Distance to home campus 
Standard deviation of professor effects 


In-person classes 0.316 0.119 0.273 0.0842 
Online classes 0.235 0.0616 0.226 0.0661 
In-person = online test p-value [0.000] [0.000] [0.000] [0.099] 


D. Dep. Var.: residuals after 2SLS with instrument = Course availability interacted with 
distance to home campus 


Standard deviation of professor effects 


In-person classes 0.312 0.0874 0.269 0.0649 
Online classes 0.236 0.0320 0.228 0.0506 
In-person = online test p-value [0.000] [0.000] [0.001] [0.239] 
Student-course observations 524,172 440,948 126,734 108,478 


Note. Each column reports estimates from a separate linear mixed model. In Panel A, in-person and online variances 
are estimated random effects parameters. The same is true for Panels B-D except that, to control for student 
selection, we first run an instrumental variables 2SLS regression, obtain the residuals, and replace the dependent 
variable in the linear mixed model with the squared residuals. Dependent variables are listed in the column headers. 
Additional fixed controls include (i) prior cumulative grade point average, (ii) an indicator for student gender, (iii) 
student age, (iv) separate fixed effects for course, term, and home campus. The estimation sample is limited to May 
2012 to November 2013, the time period in which we can identify the specific sections of courses professors taught. 
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Appendix Table A.1—Effects of Taking a Course Online Instead of In-Person on All Outcomes 


Grade: At Grade: At 


Grade: At 
least C- 


Passed 


Withdrew 


Credits 
Next 
Semester 


Enrolled 
Next 
Semester 


Credits 
Next Year 


Enrolled 
Next Year 


Online -0.380***  -0.104***  -0.110*** 


Grade least A- least B- 
Panel A. OLS Estimates 
(standard error) (0.005) (0.002) (0.002) 


Panel B. IV using Availability (current session) 


-0.092*** 
(0.001) 


-0.074*** 
(0.001) 


0.055*** 
(0.002) 


-0.801*** 
(0.056) 


-0.050*** 
(0.002) 


-0.839*** 
(0.053) 


-0.050*** 
(0.003) 


Online -0.349***  -0.109***  -0.102*** 


(standard error) (0.014) (0.006) (0.005) 
First stage coefficient -0.259 -0.259 -0.259 
First stage F-statistic 270.18 270.18 270.18 


Panel C. IV using Availability (current, next, prior sessions) 


-0.073*** 
(0.004) 
-0.259 
270.18 


-0.060*** 
(0.003) 
-0.259 
270.18 


0.044*** 
(0.004) 
-0.257 
262.58 


-0.654*** 
(0.056) 
-0.259 
215.56 


-0.030*** 
(0.004) 
-0.251 
190.81 


-1.003*** 
(0.078) 
-0.272 
263.57 


-0.030*** 
(0.006) 
-0.251 
190.81 


Online -0.380***  -0.123*** = -0.113*** 


(standard error) (0.017) (0.008) (0.006) 
First stage coefficient -0.175 -0.175 -0.175 
First stage F-statistic 161.99 161.99 161.99 


-0.083*** 

(0.005) 
-0.175 
161.99 


-0.061*** 
(0.004) 
-0.175 
161.99 


0.031*** 
(0.004) 
-0.174 
160.15 


-0.660*** 
(0.086) 
-0.175 
128.03 


-0.016** 
(0.005) 
-0.167 
115.41 


-1.004*** 
(0.110) 
-0.185 
152.24 


-0.025** 
(0.009) 
-0.167 
115.41 


Panel D. IV using Availability (current, next 1 or 2, prior 1 or 2 sessions) 


Online -0.412*** = -0.134*** = -0.123*** 
(standard error) (0.022) (0.009) (0.008) 
First stage coefficient -0.146 -0.146 -0.146 
First stage F-statistic 118.53 118.53 118.53 


-0.089*** 

(0.006) 
-0.146 
118.53 
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-0.067*** 
(0.005) 
-0.146 
118.53 


0.030*** 
(0.004) 
-0.144 
117.61 


-0.766*** 

(0.111) 
-0.144 
93.91 


-0.023*** 

(0.006) 

-0.138 
85.62 


-1.041*** 

(0.116) 
-0.154 
111.17 


-0.036*** 

(0.010) 
-0.138 
85.62 


Panel E. IV using Availability (current, next 1, 2 or 3, prior 1, 2, or 3 sessions) 


Online -0.415*** = -0.137*** = -0.125***  -0.088*** = -0.065*** 
(standard error) (0.025) (0.010) (0.008) (0.007) (0.005) 
First stage coefficient -0.132 -0.132 -0.132 -0.132 -0.132 
First stage F-statistic 100.70 100.70 100.70 100.70 100.70 
Panel F. IV using Distance to nearest campus 

Online -0.572*** — -0.131*** = -0.161*** = -0.153*** = -0.126*** 
(standard error) (0.071) (0.016) (0.020) (0.021) (0.018) 
First stage coefficient 0.001 0.001 0.001 0.001 0.001 
First stage F-statistic 19.40 19.40 19.40 19.40 19.40 


Panel G. IV using Availability (current, next, or prior sessions) interacted with distance 


Online -0.442*** = -0.123*** = -0.134***  -0.102*** = -0.084*** 
(standard error) (0.023) (0.009) (0.007) (0.007) (0.006) 
First stage coefficient 0.001 0.001 0.001 0.001 0.001 
First stage F-statistic 193.02 193.02 193.02 193.02 193.02 

N (student-by-course) 2323023 2323023 2323023 2323023 2323023 


0.029*** 
(0.004) 
-0.130 
100.28 


0.043*** 
(0.011) 

0.001 

19.93 


0.065*** 
(0.006) 

0.001 

187.83 


2601742 


-0.802*** 

(0.122) 
-0.130 
80.51 


-2.411*** 

(0.519) 
0.001 
15.75 


-0.722*** 

(0.100) 
0.001 
147.23 


1980377 


-0.017** 

(0.007) 

-0.124 
73.93 


-0.081*** 

(0.020) 
0.001 
47:13 


-0.089*** 

(0.010) 
0.001 
139.20 


2360645 


-1,099*** 

(0.132) 

-0.139 
94.19 


-3.780*** 

(0.892) 
0.000 
13.45 


-1.637*** 

(0.169) 
0.001 
146.01 


1520954 


-0.035** 

(0.011) 
-0.124 
73.93 


-0.183*** 

(0.052) 
0.001 
17.13 


-0.106*** 

(0.014) 
0.001 
139.20 


2360645 


Note. The top row of each panel reports a local average treatment effect from a separate regression, estimated using OLS (panel A) and standard instrumental 
variables methods (Panel B-G). Dependent variables are those listed in the column headers. The specifications include one endogenous treatment variable, an 
indicator = | if the student took the course online. The excluded instrument in each of Panel B-E is a binary variable = | if the course was offered in-person at the 
student’s home campus in (i) the current session, (ii) any of the current session, previous session, or following session, (iii)any of the current session, previous 
two sessions, or following two sessions, and (iv) any of the current session, previous three sessions, or following three sessions, respectively. Panel E uses 
distance from the student’s home to the nearest DeVry campus as the instrument, and panel F uses in the interaction between availability in the current, previous 
or following term and distance. Other included regressors include: (i) prior cumulative grade point average, (ii) an indicator for student gender, (iii) student age, 
and (iv) separate fixed effects for course, term, and home campus. The estimation sample is limited to those students who have address information. Standard 


errors are clustered at the section level. *p<.05, **p<.01, ***p<.001 
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Appendix Table A.2—Heterogeneity of Effect of Taking a Course Online Instead of In-person, 
by Prior Achievement, for All Outcomes 


Grade: At 
Grade least A- 
Panel A. OLS Estimates 
Online -0.570*** = -0.145*** 
(standard error) (0.013) (0.005) 
Online*Prior GPA 0.093*** = 0.020*** 
(standard error) (0.005) (0.002) 


Panel B. IV using Availability (current, next, prior sessions) 


Online -0.913***  -0,193*** 
(standard error) (0.069) (0.019) 
Online*Prior GPA 0.192*** = 0.025*** 
(standard error) (0.025) (0.007) 
First stage F-statistic: online 191.299 191.299 


First stage F-statistic: interaction 105.053 105.053 


Panel C. IV using distance 


Online -0.769*** = -0.178*** 
(standard error) (0.051) (0.014) 
Online*Prior GPA 0.134*** = 0.032*** 
(standard error) (0.016) (0.004) 


First stage F-statistic: online 571.447 571.447 
First stage F-statistic: interaction 163.017 163.017 


Grade: At 
least B- 


-0.162*** 

(0.004) 
0.025*** 

(0.002) 


-0.245*** 
(0.021) 
0.048*** 
(0.007) 
191.299 
105.053 


-0.217*** 
(0.015) 
0.038*** 
(0.005) 
571.447 
163.017 


Grade: At 
least C- 


-0.142*** 

(0.003) 
0.025*** 

(0.001) 


-0.243*** 
(0.019) 
0.060*** 
(0.007) 
191.299 
105.053 


-0.203*** 
(0.015) 
0.034*** 
(0.005) 
571.447 
163.017 


eal 


Passed 


-0.120*** 

(0.003) 
0.023*** 

(0.001) 


-0.227*** 
(0.018) 
0.060*** 
(0.006) 
191.299 
105.053 


-0.170*** 
(0.012) 
0.030*** 
(0.004) 
571.447 
163.017 


Withdrew 


0.089*** 
(0.002) 
-0.017*** 
(0.001) 


0.174*** 
(0.015) 
-0.052*** 
(0.006) 
189.787 
107.678 


0.091*** 
(0.008) 
-0.034*** 
(0.002) 
469.757 
166.666 


Credits 
Next 
Semester 


-0.733*** 
(0.064) 
-0.030* 
(0.015) 


0.508* 
(0.206) 
-0.422*** 
(0.072) 
139.888 
85.884 


-2.045*** 
(0.367) 
-0.228* 
(0.105) 
382.398 
127.671 


Enrolled 
Next 
Semester 


-0.101*** 

(0.003) 
0.026*** 

(0.001) 


-0.252*** 
(0.025) 
0.086*** 
(0.009) 
131.694 
80.627 


-0.150*** 
(0.015) 
0.050*** 
(0.003) 
372.624 
130.111 


Credits 
Next Year 


-0.903*** 

(0.055) 
0.028* 

(0.014) 


-0.654*** 
(0.135) 
-0.123*** 
(0.034) 
164.118 
90.765 


-3.055*** 
(0.565) 
-0.374* 
(0.176) 
515.331 
112.595 


Enrolled 
Next Year 


-0.116*** 

(0.005) 
0.034*** 

(0.002) 


-0.365*** 
(0.038) 
0.124*** 
(0.015) 
131.694 
80.627 


-0.253*** 
(0.039) 
0.054*** 
(0.009) 
372.624 
130.111 


Panel D. IV using interaction between availability and distance 


Online -0.621*** = -0,135*** = -0,.173*** 
(standard error) (0.023) (0.010) (0.008) 
Online*Prior GPA 0.123*** 0.008 0.027*** 
(standard error) (0.009) (0.004) (0.003) 
First stage F-statistic: online 873.301 873.301 873.301 
First stage F-statistic: interaction 98.329 98.329 98.329 

N (student-by-course) 2323023 2323023 2323023 


-0.164*** 
(0.007) 
0.043*** 
(0.002) 
873.301 
98.329 


2323023 


-0.150*** 
(0.006) 
0.046*** 
(0.002) 
873.301 
98.329 


2323023 


0.120*** 
(0.006) 
-0.040*** 
(0.002) 
895.753 
107.909 


2601742 


-0.823*** 
(0.090) 
0.066** 
(0.025) 
437.532 
90.656 


1980377 


-0.173*** 
(0.008) 
0.062*** 
(0.004) 
669.444 
88.627 


2360645 


-1.592*** 
(0.119) 
-0.025 
(0.041) 
407.763 
92.428 


1520954 


-0.211*** 
(0.010) 
0.079*** 
(0.005) 
669.444 
88.627 


2360645 


Note. Each panel reports local average treatment effects from a separate regression, estimated using OLS (panel A) and standard instrumental variables methods 
(Panel B-D). Dependent variables are those listed in the column headers. The specifications include two endogenous treatment variables, an indicator = 1 if the 
student took the course online and the indicator interacted with prior cumulative GPA. The excluded instruments are a binary variable = | if the course was 
offered in-person at the student’s home campus (defined as the nearest campus) in any of the current term, the previous term, or the following term (panel A), the 
distance in miles from the student’s home address to her home campus (panel B), the interaction of availability and distance (panel C); and the interaction of 
those variables, respectively, with prior cumulative GPA. Other included regressors include: (i) prior cumulative grade point average, (ii) an indicator for student 
gender, (iii) student age, (iv) course fixed effects, (v) session (time) fixed effects, and (vi) home campus fixed effects. The estimation sample is limited to those 
students who have address information. Standard errors are clustered at the section level. *p<.05, **p<.01, ***p<.001 
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